Method and device for generating vehicle panoramic surround view image

ABSTRACT

The present disclosure relates to a method for generating a panoramic surround view image of a vehicle, comprising: acquiring actual original images of external environment of a first part and a second part of the vehicle hinged to each other; processing the actual original images to obtain respective actual independent surround view images of the first part and the second part; obtaining coordinates of respective hinge points of the first part and the second part; determining matched feature point pairs in the actual independent surround view images of the first part and the second pan; calculating a distance between two points in each matched feature point pair accordingly, and taking matched feature point pairs with the distance less than a preset first threshold as a successfully matched feature point pairs; and taking a rotation angle corresponding to the maximum number of the successfully matched feature point pairs as a candidate rotation angle of the first part relative to the second part. The present disclosure further provides a device for generating a panoramic surround view image of a vehicle and an intelligent vehicle.

The present disclosure relates generally to the field of intelligentdriving, and more particularly, to a method and device for generating apanoramic surround view image of a vehicle.

BACKGROUND

At present, many panoramic surround view systems on the market aresuitable for vehicles in which relative position of a camera is fixedduring driving, that is, suitable for integrated vehicles. However, thiskind of system is not suitable for a split or combined vehicle(including a front of the vehicle and one or more carriages hingedtherewith) in which multiple parts are hinged to each other. The reasonis the positional relationship between the different pans hinged to eachother changes dynamically when the split vehicle is running, especiallywhen it is turning. The existing panoramic surround view solution cannotperfectly cope with this situation, and there will be display blindspots and ghosting and other issues, which could pose a security risk.

Existing methods for calibrating and stitching a panoramic surround viewsystem generally obtain rotation angles between different parts of thevehicle by installing an angle sensor device to stitch the panoramicsurround view images. However, this method requires provision of sensorsbesides the cameras, causing problems such as high cost and difficultyin installation and maintenance.

SUMMARY

In view of the technical problems existing in the prior art, the presentdisclosure provides a method for generating a panoramic surround viewimage of a vehicle, comprising: acquiring actual original images ofexternal environment of a first part and a second part of the vehiclehinged to each other; processing the actual original images to obtainrespective actual independent surround view images of the first part andthe second pan; obtaining coordinates of respective hinge points of thefirst part and the second part; determining matched feature point pairsin the actual independent surround view images of the first part and thesecond part; overlapping the respective hinge points in the actualindependent surround view images of the first part and the second part,and assuming that the independent surround view image of the first partis rotated relative to the independent surround view image of the secondpart or assuming that matched feature points of the first part arerotated relative to matched feature points of the second part,calculating a distance between two points in each matched feature pointpair accordingly, and taking a matched feature point pairs with thedistance less than a preset first threshold as a successfully matchedfeature point pair; determining an actual rotation angle of the firstpart relative to the second part at least based on number of thesuccessfully matched feature point pairs; and obtaining a panoramicsurround view image of the vehicle after fusing the respective actualindependent surround view images of the first part and the second partaccording to the coordinates of the hinge points and the actual rotationangle.

Particularly, the step of determining an actual rotation angle of thefirst part relative to the second part at least based on number of thesuccessfully matched feature point pairs comprises taking an anglecorresponding to the maximum number of the successfully matched featurepoint pairs as the actual rotation angle between the first part and thesecond part.

Particularly, the step of determining an actual rotation angle of thefirst part relative to the second part at least based on number of thesuccessfully matched feature point pairs comprises: taking an anglecorresponding to the maximum number of the successfully matched featurepoint pairs as a candidate rotation angle between the first part and thesecond part; determining coordinates of the successfully matched featurepoint pairs based on the candidate rotation angle; calculating adistance between two points in each successfully matched feature pointpair and summing the distances; and taking a rotation anglecorresponding to the minimum summation result as the actual rotationangle of the first part relative to the second part.

Particularly, the step of determining coordinates of the successfullymatched feature point pairs based on the candidate rotation anglecomprises obtaining a candidate rotation-translation matrix based on thecoordinates of the hinge points of the first part and the second partand the candidate rotation angle; and the step of calculating a distancebetween two points in each successfully matched feature point paircomprises calculating a distance between two points in the successfullymatched feature point pair based on coordinates of the matched featurepoint pair and the candidate rotation-translation matrix.

Particularly, the step of processing the actual original images toobtain respective actual independent surround view images of the firstpart and the second part comprises: correcting distortion on the actualoriginal images of the external environment of the first part and thesecond part; projecting the corrected images into the geodeticcoordinates system to generate a bird's-eye view of the first part andthe second part; detecting and matching respectively internal featurepoints in overlap-ping areas of the respective bird's-eye views of thefirst part and the second part, and then fixing and stitching to obtainrespective fixed mosaics of the first part and the second part; andcropping the respective fixed mosaics of the first part and the secondpart to obtain the respective actual independent surround view images ofthe first part and the second part.

Particularly, the step of determining matched feature point pairs in theactual independent surround view images of the first part and the secondpart comprises: feature point detection, for detecting natural featurepoints in the overlapping area of the actual independent surround viewimages of the first part and the second part and generating descriptors;feature point matching, for generating matched feature point pairs by amatching algorithm at least based on the descriptors, wherein thematching algorithm comprises ORB, SURF or SIFT algorithm; and featurepoint screening, for screening out mismatched feature point pairs by ascreening algorithm, wherein the screening algorithm comprises RANSAC orGMS algorithm.

Particularly, the method further comprises a method of obtaining thecoordinates of the respective hinge points of the first part and thesecond part that comprises: obtaining multiple pairs of trainingindependent surround view images of the first part and the second part;detecting and matching feature points in terms of each pair of thetraining independent surround view images; calculating a plurality ofcorresponding training rotation-translation matrices based on thematched feature point pairs in each pair of the training independentsurround view images, and then calculating a plurality of correspondingtraining rotation angles between the first part and the second part;determining a plurality of corresponding training translation vectorsbetween the first part and the second part at least based on coordinatesof the matched feature points in the multiple pairs of trainingindependent surround view images as well as the plurality of trainingrotation angles; and calculating to obtain the co-ordinates of the hingepoints of the first part and the second part according to thecoordinates of feature points of the multiple pairs of independentsurround view images, the plurality of training rotation angles and theplurality of training translation vectors.

Particularly, the method of obtaining the coordinates of the respectivehinge points of the first part and the second part further comprises:taking at least two of the plurality of training rotation angles as oneset, and calculating to obtain coordinates of candidate hinge pointscorresponding to the set based on the training translation vectors; andsorting all the coordinates of candidate hinge points, and taking amedian of the sorting result as the coordinates of the hinge points ofthe first part and the second part; wherein a difference between atleast two training rotation angles of each set is greater than a presetangle threshold.

The present disclosure also relates to a device for generating apanoramic surround view image of a vehicle, the device comprising: anoriginal image acquisition unit configured to acquire actual or trainingoriginal images of external environment of a first part and a secondpart of the vehicle hinged to each other; an independent surround viewimage acquisition unit, which is coupled to the original imageacquisition unit, configured to stitch the actual or training originalimages of the first part and the second part into respective actual ortraining independent surround view images; and a panoramic surround viewimage acquisition unit coupled to a hinge point calibration unit and theindependent surround view image acquisition unit, comprising: a featurepoint detecting and matching module, which is coupled to the independentsurround view image acquisition unit, configured to receive the actualindependent surround view images of the first part and the second part,and detect and match feature points therein; an actual rotation anglecalculation module, which is coupled to the feature point detecting andmatching module, con-figured to obtain coordinates of hinge points ofthe first part and the second part, and overlap the hinge points of thefirst part and the second part in the independent surround view images,and, assuming that the independent surround view image of the first partis rotated relative to the independent surround view image of the secondpart or that the matched feature points of the first part are rotatedrelative to the matched feature points of the second part, calculate adistance between two points in each matched feature point pair to takethe matched feature point pairs with the distance less than a presetfirst threshold as a successfully matched feature point pairs, anddetermine an actual rotation angle of the first part relative to thesecond part at least based on number of the successfully matched featurepoint pairs; and a panoramic surround view image generation module,which is coupled to the actual rotation angle calculation module,configured to obtain a panoramic surround view image of the vehicleafter fusing the respective actual independent surround view images ofthe first part and the second part according to the coordinates of thehinge points and the actual rotation angle.

Particularly, the step of the actual rotation angle calculation moduleconfigured to determine the actual rotation angle of the first partrelative to the second part at least based on the number of thesuccessfully matched feature point pairs comprises, taking an anglecorresponding to the maximum number of the successfully matched featurepoint pairs as the actual rotation angle between the first part and thesecond part.

Particularly, the step of the actual rotation angle calculation moduleconfigured to determine the actual rotation angle of the first partrelative to the second part at least based on the number of thesuccessfully matched feature point pairs comprises, taking an anglecorresponding to the maximum number of the successfully matched featurepoint pairs as a candidate rotation angle between the first part and thesecond part, determining coordinates of the successfully matched featurepoint pair based on the candidate rotation angle, calculating a distancebetween two points in each successfully matched feature point pair andsumming the distances, and taking a rotation angle corresponding to theminimum summation result as the actual rotation angle of the first partrelative to the second part.

Particularly, the device further comprises the hinge point calibrationunit coupled to the independent surround view image acquisition unit,which comprises: a feature point detecting and matching module, which iscoupled to the independent surround view image acquisition unit,configured to receive multiple pairs of training independent surroundview images of the first part and second part, and to detect and matchfeature points of each pair of training independent surround view imagesof the first part and the second part; a training rotation anglecalculation module, which is coupled to the feature point detecting andmatching module, configured to obtain a plurality of trainingrotation-translation matrices between the feature points of the firstpart and the second part in each pair of training independent surroundview images based on the coordinates of the matched feature points, andcorrespondingly obtain a plurality of training rotation angles betweenthe first part and the second part in each pair of independent surroundview images; a training translation vector acquisition module, which iscoupled to the training rotation angle calculation module, configured todetermine a plurality of corresponding training translation vectors ofeach pair of training independent surround view images according tocoordinates of the matched feature points of each pair of trainingindependent surround view images and the plurality of training rotationangles; and a hinge point coordinates determination module, which iscoupled to the translation vector acquisition module and the trainingrotation angle calculation module, configured to determine thecoordinates of the hinge points of the first part and the second part ofthe vehicle according to the coordinates of the matched feature pointsof the multiple pairs of training independent surround view images, theplurality of training rotation angles and the corresponding plurality oftraining translation vectors.

The present disclosure further relates to an intelligent vehicle,comprising: a first part and a second part hinged to each other; aprocessor, and a memory coupled to the processor; and a sensor unitconfigured to capture actual or training original images of the firstpart and the second part; wherein the processor is configured toimplement the method as described above.

BRIEF DESCRIPTION OF THE DRAWINGS

Preferred embodiments of the present disclosure will be describedhereinafter with further details with reference to the accompanyingdrawings, wherein:

FIG. 1 is a schematic diagram of a vehicle according to an embodiment ofthe present disclosure;

FIG. 2A is an overall flow diagram of a method for generating apanoramic surround view image of a vehicle according to an embodiment ofthe present disclosure;

FIG. 2B is a specific flow diagram of a method for generating apanoramic surround view image of a vehicle according to an embodiment ofthe present disclosure;

FIG. 3 is a flow diagram of a method for fixing and stitching actualoriginal images of various hinged parts of a vehicle according to anembodiment of the present disclosure;

FIG. 4 is a flow diagram of a method for calculating the coordinates ofhinge points of various parts of a vehicle according to an embodiment ofthe present disclosure;

FIG. 5 is a schematic diagram of a device for generating a panoramicsurround view image of a vehicle according to an embodiment of thepresent disclosure;

FIG. 6 is a schematic diagram of an independent surround view imageacquisition unit of the device for generating a panoramic surround viewimage of a vehicle according to an embodiment of the present disclosure;

FIG. 7 is a schematic diagram of a hinge point calibration unit of thedevice for generating a panoramic surround view image of a vehicleaccording to an embodiment of the present disclosure;

FIG. 8 is a schematic diagram of a panoramic surround view imageacquisition unit of a device for generating a panoramic surround viewimage of a vehicle according to an embodiment of the present disclosure;and

FIG. 9 is a schematic diagram of an intelligent vehicle according to anembodiment of the present disclosure.

DETAILED DESCRIPTION

In order to state the object, technical solutions and advantages of theembodiments of the present disclosure more clearly, the technicalsolutions in the embodiments of the present disclosure will be describedclearly and completely below with reference to the drawings of theembodiments of the present disclosure. Obviously, the describedembodiments are only part of the embodiments of the present disclosure,rather than the overall. Based on the embodiments in the presentdisclosure, all other embodiments obtained by those ordinarily skilledin the art without creative efforts shall fall within the protectionscope of the present disclosure.

In the following detailed description, reference may be made to theaccompanying drawings, which are considered as a part of the presentdisclosure to illustrate specific embodiments of the present disclosure.In the drawings, similar reference signs describe substantially similarcomponents in the different drawings. The specific embodiments of thepresent disclosure are described in sufficient detail below to enablethose ordinarily skilled in the art to implement the technical solutionsof the present disclosure. It should be appreciated that otherembodiments may also be utilized or structural, logical or electricalchanges may be made to the embodiments of the present disclosure.

The present disclosure provides a method for generating a panoramicsurround view image of a vehicle, wherein the vehicle comprises at leasttwo parts that are hinged together, such as a semi-trailer. In someembodiments, the vehicle comprises multiple parts that are hinged witheach other, such as trains, subways, or the like, to which the methoddescribed in the present disclosure is also applicable.

Structure of the vehicle applied with the method will be describedbriefly below by taking a vehicle comprising two parts hinged to eachother as an example. FIG. 1 is a schematic diagram of a vehicleaccording to an embodiment of the present disclosure. In thisembodiment, it is assumed that the left side in the figure is theforward direction of the vehicle. The vehicle shown in FIG. 1 includes afirst part 101 and a second part 102, which are connected by a hinge. Inthe hinged state, hinge points of the first part 101 and the second part102 coincide with each other. Cameras 11, 12 and 13 are respectivelydisposed on the front, right and left sides of the first part 101.Cameras 21, 22, and 23 are provided respectively on the rear, right, andleft sides of the second part 102. In this embodiment, the cameras maybe wide-angle cameras of 180° or other angles, and the arrangement ofthe cameras is only an example. In some embodiments, the position andamount of the cameras could be set in other ways.

In some embodiments, the vehicle may further include a camerasynchronization module (not shown) to synchronize data among the variouscameras.

For a vehicle including a plurality of parts hinged to each other, hingepoints are irreplaceable with respect to other features. The reason isthat no matter how state of motion of the two hinged parts of thevehicle changes, the respective hinge points of the two parts of thevehicle should always coincide in actual situations. Taking such a pointrelatively stable in position as a reference point can not only make thecalculation result more accurate and generated images closer to theactual scene, but also reduce extra computational power loss caused byrelative motion and thereby improve the overall work efficiency.Therefore, coordinates of the respective hinge points of the hingedparts are calculated first, and then used in the subsequent imagestitching process based on the known coincidence relationship, by whicha more realistic and stable panoramic surround view image of the vehiclecan be obtained.

The flow of the method in the present disclosure will be described indetail below. FIG. 2A is an overall flow diagram of a method forgenerating a panoramic surround view image of a vehicle according to anembodiment of the present disclosure, and FIG. 2B is a specific flowdiagram of a method for generating a panoramic surround view image of avehicle according to an embodiment of the present disclosure. In someembodiments, as shown in FIGS. 2A and 2B, the method for generating apanoramic surround view image of a vehicle described in the presentdisclosure can be summarized into four sections: acquiring actualoriginal images 21, acquiring actual independent surround view images22, acquiring coordinates of hinge points 23 and acquiring a panoramicsurround view image 24.

The so-called actual original images and actual independent surroundview images here are to be distinguished from the training originalimages and the training actual independent surround view images in theprocess of calculating the coordinates of the hinge points. The “actual”emphasizes the images obtained during the actual driving process of thevehicle. The “training” emphasizes the images obtained by artificiallysetting a specific angle between the first part and the second part ofthe vehicle in order to obtain the coordinates of the hinge points.

At Step 21, the operation of acquiring actual original images include:

Step 201, acquiring actual original images of external environment ofthe first part and the second part of the vehicle hinged to each other,wherein the first part and the second part referred to herein may be,for example, a front and a carriage, which of course could be twocarriages in other cases.

The actual original images are the images directly obtained by thecameras, and they may include the actual original images of the externalenvironment of the first part and second part. Of course, these actualoriginal images may also include partial images of the vehicle itself.In some embodiments, wide-angle cameras can be provided outside thevehicle to obtain the actual original images. In some embodiments, thewide-angle cameras may be cameras of 180° or other angles. In someembodiments, shooting angle of each actual original image may beenlarged as much as possible in order to obtain better imagingperformances.

In Step 22, the operation of acquiring actual independent surround viewimages include:

Step 202, processing the actual original images to obtain respectiveactual independent surround view images of the first part and the secondpart.

Data of the actual original images of the first part and the second partobtained in the Step 201 are unprocessed, and the images captured byadjacent cameras have overlapping areas. Therefore, it is necessary toconvert the actual original images (the specific conversion process willbe described in details later), and then fix and stitch the multipleimages belonging to the same part (the first part or the second part) toobtain a complete actual independent surround view image of each part.The actual independent surround view image of the first or second partis a complete top view of the actual external environment of the partexcept the hinged side.

Before continuing to introduce other operations for obtaining thepanoramic surround view image, a specific method for obtaining theindependent surround view image of each part will be introduced indetails first. FIG. 3 is a flow diagram of a method for fixing andstitching actual original images of various hinged parts of a vehicleaccording to an embodiment of the present disclosure, which actually isfurther description of the Step 202 in the foregoing method. The methodincludes:

Step 301, correcting distortion on the actual original images.

In some embodiments, the original images captured by the wide-anglecameras have a feature of perspective distortion, the effect of whichwill cause the images to be distorted and thus cannot correctly reflectthe distance relationship of objects in the images. In order toeliminate this distortion, it is necessary to perform distortioncorrection processing on the original images collected by the wide-anglecameras. In some embodiments, the camera parameters calibrated by thewide-angle cameras and distortion correction parameters may be used toperform correction processing on the actual original images to obtaincorrected images corresponding to the actual original images. The cameraparameters and distortion correction parameters could be determinedaccording to the internal structure of the wide-angle cameras and anestablished distortion model.

Step 302, performing perspective transformation on the distortioncorrected images.

In practical applications, what a user needs to see is the operatingstate diagram of the vehicle from a top perspective, so it is alsonecessary to perform perspective transformation on the distortioncorrected images. In some embodiments, the images obtained by differentcameras are projected into the geodetic coordinates system to obtain anactual bird's-eye view (which can be obtained by selecting the specifiedfeature points of the calibration object and performing perspectivetransformation), and generate a mapping relationship between thecorrected images to the actual bird's-eye view, to obtain an actualbird's-eye view corresponding to each corrected image.

Step 303, fixing and stitching the actual bird's-eye views.

For all the parts of the vehicle hinged to each other, the actualbird's-eye views of each part in all directions can be stitchedtogether. Due to the characteristics of wide-angle cameras, there is apartial overlapping area between the actual bird's-eye views captured byadjacent cameras on each part, and thus the overlapping area needs to befixed to stitch an actual fixed mosaic.

In some embodiments, fixing and stitching can be achieved by manuallyselecting several marker points for matching. Of course, other knownmethods can also be used for the matching.

Step 304, cropping the actual fixed mosaic.

The actual fixed mosaic of each part of the vehicle is obtained in theforegoing steps, but such an actual fixed mosaic may also includeunnecessary portions. In some embodiments, for the actual fixed mosaicof each part, the region of interest can be cropped as required, so thatthe image size conforms to the display range of the screen, therebybeing displayed on the screen, and finally the respective actualindependent surround view images of the first part and the second partcan be obtained.

What follows is introduction of the method for generating a panoramicsurround view image of a vehicle described in the present disclosure asshown in FIGS. 2A and 2B. In Step 23, the step of acquiring thecoordinates of the hinge points specifically includes:

Step 203, obtaining coordinates of hinge points.

For the vehicle, the coordinates of the hinge points can be calculatedeach time when it is started and initialized, or the hinge points can becalibrated at the same time when the cameras coordinates are calibrated.During the driving, the calculation can be performed according to thecoordinates of the known hinge points, and it is not necessary to makerepeated calculation. The method for calculating the coordinates of thehinge points will be introduced separately later.

The so-called training independent surround view image here refers tothat, in order to calculate the hinge point coordinates of the firstpart and the second part, n kinds of relative positions are artificiallyformed between the first part and the second part, and n pairs oftraining independent surround view images of the first part and thesecond part are obtained accordingly. Based on the matched featurepoints in the n pairs of training independent surround view images, ntraining rotation angles can be obtained accordingly by calculation.Combined with the corresponding n training translation vectors, therespective coordinates of hinge points of the first part and second partof the vehicle can be determined.

FIG. 4 is a flow diagram of a method for calculating the coordinates ofhinge points of various parts of a vehicle according to an embodiment ofthe present disclosure. As described above, the calculation of thecoordinates of hinge points is not part of the method shown in FIG. 2A,but a method of calculating the coordinates of hinge points that hasbeen performed before the method.

As shown in FIG. 4 , the method for calculating the coordinates of thehinge points of each part of the vehicle may include:

In step 401, obtaining respectively n pairs of training original imagesof the first part and the second part, and fixing and stitching toobtain n pairs of training independent surround view images. Each pairof images corresponds to a relative position of the first part and thesecond part, and there are n kinds of positions in total, wherein n canbe a positive integer greater than or equal to 2. The operation ofobtaining training independent surround view images by fixed stitchingis similar to the operation of obtaining actual independent surroundview images described above, which will not be described again.

Step 402, detecting and matching feature points of each of the n pairsof training independent surround view images of the first part and thesecond part.

In some embodiments, a feature point refers to a point in the image thathas distinct characteristics and can effectively reflect the essentialcharacteristics of the image and can identify the target object in theimage. The feature points of an image comprise two sections: keypointand descriptor. The keypoint refers to the position, orientation andscale information of the feature point in the image. The descriptor isusually a vector, which describes information of the pixels around thekeypoint as artificial design approach. Generally, similar-lookingkeypoints have corresponding similar descriptors. Therefore, during thematching, two feature points could be deemed as the same feature pointas long as their descriptors are similar in distance in a vector space.

Specifically, for each pair of training independent surround viewimages, keypoints of the training independent surround view images ofthe two hinged parts can be obtained, and the descriptors of the featurepoints can be calculated according to the positions of the keypoints.The feature points of the surround view images of the two hinged partsof the vehicle are matched according to the descriptors of the featurepoints to obtain a matched feature point pair of the surround viewimages of the two hinged parts of the vehicle. In one embodiment, theBrute Force matching algorithm can be used, which compares thedescriptors of the feature points of the training independent surroundview images of the two hinged parts one by one in a vector space, andselects the pair with a smaller distance as a matched point pair.

Step 403, calculating n training rotation-translation matrices betweenthe feature points of the first part and the second part based on thematched feature point pair, and calculating n training rotation anglesbetween the first part and the second part accordingly. Here are someexamples of calculating training rotation angles based on the matchedfeature point pair.

In some embodiments, the random sampling consensus algorithm (RANSAC) orthe least median method (LMedS) can be used to select matched featurepoint pairs of the training independent surround view images of the twohinged parts. Taking the random sampling consensus algorithm as anexample, specifically, several pairs of matched points are extractedfrom the obtained matched point pairs, a rotation-translation matrix iscalculated, and the pairs of matched points are recorded as “interiorpoints”. Searching for non-interior points in the matched point pairs isthen continued, and the matched point pairs, if fitting the matrix, willbe added as the interior points. When the number of the point pairs inthe interior points is greater than a preset threshold, therotation-translation matrix can be determined from these data. Accordingto the above method, randomly sampling is conducted k times (k is apositive integer greater than 0) to select the largest set of interiorpoints, and eliminate mismatched point pairs such as the non-interiorpoints. Only after the mismatched points are eliminated, the correctmatched point pairs in the interior points can be used to obtain thetraining rotation-translation matrix corresponding to a specificposition. According to n training rotation-translation matrices, ntraining rotation angles θ₁ . . . θ_(n) between the first part and thesecond part are obtained.

In the method involved in the present disclosure, the acquisition of thetraining rotation angles is obtained by calculation. Different from themethod through physical measurement in the prior art, the methodinvolved in the present disclosure could obtain a more accurate result,and the operation difficulty of obtaining the training rotation anglesis lower. Meanwhile, the method in the present disclosure reducesprovision of sensors, causing a lower cost and wider applicability, andit can avoid interference factors in the environment.

Step 404, determining n training translation vectors at least based onthe coordinates of the matched feature points between the n pairs oftraining independent surround view images of the first part and thesecond part and the corresponding n training rotation angles between thefirst part and the second part.

According to one embodiment, suppose (a_(x), a_(y)) can be set as thecoordinates of the hinge point of the first part 101 of the vehicle,(b_(x), b_(y)) as the coordinates of the hinge point of the second part102 of the vehicle, (x₀, y₀) and (x₁, y₁) are the coordinates of thefeature points matching each other in the training independent surroundview images of the first part 101 and the second part 102 respectively,and θ is the training rotation angle of the first part and the secondpart, then

$\begin{matrix}{{{\begin{bmatrix}{\cos\theta} & {{- \sin}\theta} & 0 \\{\sin\theta} & {\cos\theta} & 0 \\0 & 0 & 0\end{bmatrix}\begin{bmatrix}{x_{0} - a_{x}} \\{y_{0} - a_{y}} \\1\end{bmatrix}} + \begin{bmatrix}b_{x} \\b_{y} \\1\end{bmatrix}} = \begin{bmatrix}x_{1} \\y_{1} \\1\end{bmatrix}} & (1)\end{matrix}$

The formula (1) can be disassembled to obtain the formula (2)

$\begin{matrix}\left\{ \begin{matrix}{{{\cos\theta*x_{0}} - {\sin\theta*y_{0}} - {a_{x}*\cos\theta} + {a_{y}*\sin\theta} + b_{x}} = x_{1}} \\{{{\sin\theta*x_{0}} + {\cos\theta*y_{0}} - {a_{x}*\sin\theta} - {a_{y}*\cos\theta} + b_{y}} = y_{1}}\end{matrix} \right. & (2)\end{matrix}$

The training translation vector is the training translation parameter ofthe feature points of the training independent surround view images ofthe two hinged parts translating from one image to the other image, forexample, the training translation parameter of the feature points in thematched point pairs translating from the independent surround view imageof the first part 101 to the independent surround view image of thesecond part 102. Therefore, for a pair of matched points, assuming thatthe coordinates of the feature point of the training independentsurround view image of the first part 101 of the vehicle is the origin,the coordinates of the matched point corresponding to the trainingindependent surround view image of the second part 102 of the vehicle isnumerically equal to the training translation vector for the two images.That is, if the feature point of the training surround view image of thefirst part (x₀, y₀) is set as (0, 0), the coordinates of the featurepoint matched by the training surround view image of the second part(x₁, y₁) is the training translation vector from the training surroundview image of the first part to the training surround view image of thesecond part (dx, dy), which can be expressed by formula (3):

$\begin{matrix}\left\{ \begin{matrix}{{{{- a_{x}}*\cos\theta} + {a_{y}*\sin\theta} + b_{x}} = {dx}} \\{{{{- a_{x}}*\sin\theta} - {a_{y}*\cos\theta} + b_{y}} = {dy}}\end{matrix} \right. & (3)\end{matrix}$

Step 405, calculating to obtain the coordinates of the hinge point ofthe first part and the second part of the vehicle according to thecoordinates of the feature points, the training rotation angles and thetraining translation vectors of the n pairs of training independentsurround view images. The calculation here is based on such a premisethat the points in the training independent surround view images of thefirst part and the second part of the vehicle are all rotated aroundtheir respective hinge points when moving direction of the vehiclechanges.

For each training rotation angle θ, there is a corresponding trainingtranslation vector. (a_(x), a_(y), b_(x), b_(y)) can be obtained bybringing then training rotation angles and n training translationvectors into formula (4):

$\begin{matrix}{\arg\min\limits_{({a_{x},a_{y},b_{x},b_{y}})}{{{\begin{bmatrix}{{- \cos}\theta_{1}} & {\sin\theta_{1}} & 1 & 0 \\{\sin\theta_{1}} & {{- \cos}\theta_{1}} & 0 & 1 \\ \vdots & \vdots & \vdots & \vdots \\{{- \cos}\theta_{n}} & {\sin\theta_{n}} & 1 & 0 \\{\sin\theta_{n}} & {{- \cos}\theta_{n}} & 0 & 1\end{bmatrix} \star \begin{bmatrix}\begin{matrix}\begin{matrix}a_{x} \\a_{y}\end{matrix} \\b_{x}\end{matrix} \\b_{y}\end{bmatrix}} - \begin{bmatrix}\begin{matrix}\begin{matrix}\begin{matrix}{dx}_{1} \\{dy}_{1}\end{matrix} \\ \vdots \end{matrix} \\{dx}_{n}\end{matrix} \\{dy}_{n}\end{bmatrix}}}} & (4)\end{matrix}$

In general, multiple pairs of training independent surround view imagesof the first part and second part should be taken for test calculation,and thus the value of n will be much larger than 2, thereby forming anoverdetermined system such as the formula (4). The least squares methodcan be used to solve the overdetermined system, and the coordinates ofhinge point of the first part and the second part that are closest tothe actual value can be obtained by the calculation, that is, (a_(x),a_(y), b_(x), b_(y)) with the smallest value of the expression.

Considering that if there are several outliers in the above n angles,they will greatly affect calculation results of the coordinates of thehinge point. Accordingly, a further method can be used to eliminate theinterference caused by outliers.

According to one embodiment, it is assumed that two sets of trainingrotation angles (θ_(i), θ_(j)) and corresponding training translationvectors are selected from the n training rotation angles. The selectedtraining rotation angles should satisfy |θ_(i)−θ_(j)|>ξ (where ξ is apreset angle, which can be set upon actual requirements; for example, itas large as possible can be selected to satisfy the accuracy of thecalculation, i and j are integers greater than 0 and less than or equalto n, and i is not equal to j), in order to make the obtained resultmore robust. That is, the angle difference between any two trainingrotation angles is greater than the preset value. Therefore, for any setof training rotation angles (θ_(i), θ_(j)) and training translationparameters (dx_(i), dy_(i), dx_(j), dy_(j)), a set of coordinates of thecandidate hinge point coordinates (a_(x), a_(y), b_(x), b_(y)) isobtained by solving the formula (5):

$\begin{matrix}{\arg\min\limits_{({a_{x},a_{y},b_{x},b_{y}})}{{{\begin{bmatrix}{{- \cos}\theta_{i}} & {\sin\theta_{i}} & 1 & 0 \\{\sin\theta_{i}} & {{- \cos}\theta_{i}} & 0 & 1 \\{{- \cos}\theta_{j}} & {\sin\theta_{j}} & 1 & 0 \\{\sin\theta_{j}} & {{- \cos}\theta_{j}} & 0 & 1\end{bmatrix} \star \begin{bmatrix}\begin{matrix}\begin{matrix}a_{x} \\a_{y}\end{matrix} \\b_{x}\end{matrix} \\b_{y}\end{bmatrix}} - \begin{bmatrix}\begin{matrix}\begin{matrix}{dx}_{i} \\{dy}_{i}\end{matrix} \\{dx}_{j}\end{matrix} \\{dy}_{j}\end{bmatrix}}}} & (5)\end{matrix}$

Then with m sets of training rotation angles (θ_(i), θ_(j)) and m setsof training translation parameters (dx_(i), dy_(i), dx_(j), dy_(j)) thecoordinates of m sets of candidate hinge points can be obtained:

$\begin{matrix}\begin{bmatrix}\begin{matrix}\begin{matrix}\left( {a_{x},a_{y},b_{x},b_{y}} \right)_{1} \\\left( {a_{x},a_{y},b_{x},b_{y}} \right)_{2}\end{matrix} \\ \vdots \end{matrix} \\\left( {a_{x},a_{y},b_{x},b_{y}} \right)_{m}\end{bmatrix} & (6)\end{matrix}$

Then the coordinates of the m sets of candidate hinge points are sorted,the sorting result showing a Gaussian distribution, and a median of thesorting results is taken as the coordinates of the hinge point.

In this embodiment, n could be an integer greater than or equal to 2, mcould be an integer greater than or equal to 1, and m is less than n.With the above method, the influence of the outliers of the trainingrotation angles on the calculation result can be effectively reduced.

In the above method, the final coordinates of the hinge point areobtained by matching and calculating the feature points of the trainingindependent surround view images of the two hinged parts. Compared withthe traditional physical measurement technology, the coordinates of thehinge point obtained by this method are more accurate. For vehicles withhinged structures, it is more adaptable because there is no need toinstall physical measurement instruments. The method is simple andreliable in operation, and can achieve calibration of the hinge pointwithout the aid of other tools, saving labor cost and materialresources.

As shown in FIGS. 2A and 2B, what follows is the introduction of themethod for generating a panoramic surround view image of a vehicledescribed in the present disclosure. At Step 24, the operation ofacquiring a panoramic surround view image includes:

Step 204, determining matched feature point pairs in the overlappingarea of the actual independent surround view images of the first partand the second part. The matching here means that there is acorresponding relationship between the two points, or the same point isrepresented in the independent surround view images of the first partand the second part. However, in practical situations, the matchedpoints are not necessarily the points that are successfully matched asmentioned later.

In some embodiments, the method for matching the feature point pairsincludes feature point detection, feature point matching, and featurepoint screening. In some embodiments, the foregoing method process issimilar to Step 402 and Step 403, which would not be repeated here, andthe difference is that the image data of the operation are imagesobtained during actual driving instead of training images.

In some embodiments, the feature point detection method may include ORB,SURF, or SIFT algorithm, or the like.

In some embodiments, the feature point matching algorithm may includethe Brute Force matching algorithm or the Nearest Neighbor matchingalgorithm, or the like.

In some embodiments, the feature point screening method is furtherincluded, which may include RANSAC or GMS algorithm or the like.

Step 205, overlapping the hinge points of the actual independentsurround view images of the first part and the second part, and assumingthat the independent surround view image of the second part remainsstationary, rotating the independent surround view image of the firstpart around the overlapped hinge points, or assuming the matched featurepoints of the first part are rotated relative to the matched featurepoints of the second part, determining the number of successfullymatched point pairs in the matched feature point pairs.

Of course, according to other embodiments, it can also be assumed thatthe independent surround view image of the first part remainsstationary, the independent surround view image of the second part isrotated around the hinge points, or that the matched feature points ofthe second part rotate relative to the matched feature points of thefirst part.

According to an embodiment, the coordinates of the feature points in theactual independent surround view image of the first part of the vehicleare set as x_(i), and the coordinates of the feature pointscorresponding to x_(i) in the actual independent surround view image ofthe second part are x′_(i), and ε is the distance between the matchedfeature point pairs in the actual independent surround view images ofthe first part and the second part, as expressed in formula (7):

ε=∥x′ _(i) −H*x _(i)∥  (7)

where H(β, t) is the rotation-translation matrix (two-dimensional)between the feature point sets matched by the actual independentsurround view images of the first part and the second part, as expressedby formula (8). β is the hypothetical rotation angle between the firstpart and second part of the vehicle. t(t₁, t₂) is the translation vectorof the hinge point of the independent surround view image of the firstpart relative to the hinge point of the independent surround view imageof the second part.

$\begin{matrix}{{H\left( {\beta,t} \right)} = \begin{bmatrix}{\cos\beta} & {{- \sin}\beta} & t_{1} \\{\sin\beta} & {\cos\beta} & t_{2}\end{bmatrix}} & (8)\end{matrix}$

According to one embodiment, when the distance ε of the matched featurepoint pairs detected in the actual independent surround view images ofthe first part and the second part is smaller than a first threshold σ,it is considered that the feature point matching is successful.According to one embodiment, the first threshold may be set upon user'sneeds.

In some embodiments, this result can be expressed using the Iversonbracket, as shown in formula (9):

$\begin{matrix}{\lbrack P\rbrack = \left\{ \begin{matrix}1 & {{if}P{is}{true}} \\0 & {otherwise}\end{matrix} \right.} & (9)\end{matrix}$

where P can be expressed as formula (10):

∥x′ _(i) −H*x _(i)∥<σ  (10)

That is to say, when the value on the left side of the formula (10) isless than the first threshold σ, P is 1, otherwise P is 0.

Formula (11) can be used to calculate the number k of successfullymatched feature point pairs

k=Σ _(i=0) ^(L)[∥x′ _(i) −H*x _(i)∥<σ]  (11)

The value of i ranges from 0 to L, where L is the number of matchedfeature point pairs in the actual independent surround view images ofthe first part and the second part, and L is an integer greater than orequal to 1.

Step 206, taking the rotation angle corresponding to the maximum numberof successfully matched feature point pairs as the candidate rotationangle of the first part relative to the second part.

According to an embodiment, formula (12) can be used to determine thecandidate rotation angle β₁ corresponding to the maximum number ofsuccessfully matched feature point pairs:

β₁=argmax Σ_(i=0) ^(L)[∥x′ _(i) −H*x _(i)∥<σ]  (12)

In this embodiment, the candidate rotation angle is unique; if it is notunique, the calculation can be continued according to the followingembodiment to determine the actual rotation angle.

Optionally, according to an embodiment, Step 206 may directly jump toStep 208, where is taking the candidate rotation angle as the actualrotation angle, fusing the respective actual independent surround viewimages of the first part and the second part according to thecoordinates of the hinge points and the actual rotation angle, to obtaina panoramic surround view image of the vehicle.

In the method introduced in this embodiment, the actual rotation anglebetween the first part and the second part of the vehicle can be quicklydetermined, which saves computing resources and facilitates rapidsynthesis of a panoramic surround view image of the vehicle duringactual driving.

Optionally, in another embodiment, Step 206 could jump to Step 207.

Step 207, where is determining coordinates of the successfully matchedfeature point pair based on the candidate rotation angle, calculatingdistances between two points in each successfully matched feature pointpair and summing the distances, and taking the rotation anglecorresponding to the minimum summation result as the actual rotationangle of the first part relative to the second part.

If there is only one candidate rotation angle obtained in Step 206, Step207 can still be used to fine-tune the candidate rotation angle. If theoperation of Step 207 is directly performed without going through Step206, a wrong rotation angle may be obtained because the sum of thedistances between the mismatched feature point pairs is the smallest.Therefore, it is important to perform the operation in Step 206 first.

Formula (13) can be used to determine the candidate rotation angle β₂corresponding to the minimum sum of the distances between feature pointpairs:

β₂=argmin Σ_(j=0) ^(G)∥x′ _(j) −H*x _(j) ∥,x _(j) ,x′ _(j) ∈X _(G)  (13)

where G is the number of successfully matched feature point pairs, X_(G)is the set of the successfully matched feature points, G is an integergreater than or equal to 1, and G is less than or equal to L.

If the number of the candidate rotation angles obtained in Step 206 isnot unique, it is necessary to select one therefrom as the final actualrotation angle. Suppose there are F candidate rotation angles, where Fis an integer greater than 1. The F rotation angles could be broughtinto the formula (8) to obtain F candidate rotation-translation matricesH, and the F rotation-translation matrices are brought into the formula(10) to calculate the coordinates of all the successfully points Xj andXj′, obtaining a point set X_(G).

Then, formula (14) is used to calculate the angle β₂ corresponding tothe minimum sum of distances between successfully matched feature pointpairs based on multiple candidate rotation-translation matrices H_(m).

β₂=argmin Σ_(m=0) ^(F)Σ_(j=0) ^(G) ∥x′ _(j) −H _(m) *x _(j) ∥,x _(j) ,x′_(j) ∈X _(G)  (14)

After the candidate rotation angle β₂ corresponding to the minimum sumof the distances between the successfully matched feature point pairs isdetermined, it can be used as the actual rotation angle of the firstpart relative to the second part.

Step 207 can directly jump to Step 209, where according to thecoordinates of the hinge point and the actual rotation angle β₂, theactual independent surround view images of the first part and the secondpart are fused to obtain a panoramic surround view image of the vehicle.

The method introduced in this embodiment further sets constraints forfine calculation, and further fine-tunes the calculation results in Step206, so that the successfully matched feature point pairs are closer toeach other, bringing about higher degree of coincidence, and theobtained actual rotation angle is closer to reality, which improvescertainty of the actual rotation angle obtained by the calculation andmakes the final result more robust and more accurate.

In addition, the method in this embodiment also solves the problem thatthe multiple candidate rotation angle cannot be determined since thereare multiple candidate rotation angles obtained in Step 206, therebyimproving certainty of the final result.

The method of the present disclosure is based on the hinge relationshipof the first part and the second part, thus the precondition of thecoincidence of the coordinates of the hinge points of the two parts isset so that the angle with the largest number of successfully matchedfeature point pairs is selected as the actual rotation angle under thisprecondition. However, in the existing methods, calculating therotation-translation relationship between the first part and the secondpart only based on the matched feature points in the surround view imagewould cause a peristaltic visual effect that the distance between thefirst part and the second part seems sometimes long and sometimes shortin different situations. The present disclosure avoids this problem verywell because the coordinates of the hinge point of the first part andthe second part are fixed.

In the method of the present disclosure, there is no need to installother sensors (such as angle sensors) to obtain vehicle steering data,and the panoramic surround view function in the case of the angle of thevehicle is changing can be achieved only through the images collected bythe cameras and the feature matching of images, which solves the problemof the traditional surround view solutions that seamless surround viewstitching cannot be achieved in the case of real time change of therotation angles of the first part and the second part, thereby bringingabout the advantages of simple operation and low cost.

FIG. 5 is a schematic diagram of a device for generating a panoramicsurround view image of a vehicle according to an embodiment of thepresent disclosure. This device may be located on the vehicle or in aremote server of the vehicle.

As shown in the figure, the structure of the device for generating apanoramic surround view image of a vehicle may include:

An original image acquisition unit 501 configured to acquire an originalimage of external environment of two hinged parts of the vehicle. Insome embodiments, the original image acquisition unit 501 may includeone or more wide-angle cameras disposed on the non-hinged side of thetwo hinged parts of the vehicle. The original image acquisition unit 501can be used to acquire actual original images and can also be used toacquire training original images.

An independent surround view image acquisition unit 502, which iscoupled to the original image acquisition unit 501, configured to stitchthe respective original image data of the two parts into an independentsurround view image of the part. The independent surround view imageacquisition unit 502 can be used to obtain actual independent surroundview images of the first part and the second part of the vehicle and canalso be used to obtain training independent surround view images of thetwo parts.

A hinge point calibration unit 503, which is coupled to the independentsurround view image acquisition unit 502, configured to calculate thecoordinates of hinge points according to the training independentsurround view image data. According to one embodiment, the hinge pointcalibration unit 503 could be configured to perform the operation in theSteps 402-405 of the method in the embodiment shown in FIG. 4 . Theoperation of this unit is to obtain the coordinates of hinge points. Thecoordinates of hinge points are obtained by calculation before actualdriving of the vehicle or at each time when it is started. Onceobtained, the coordinates value is recorded for subsequent use.

Of course, according to other embodiments, the device may also notinclude a hinge point calibration unit, and the coordinates of hingepoints may be calculated by a remote server, in which the calculatedcoordinates of hinge points may be sent to the device.

A panoramic surround view image acquisition unit 504, which is coupledto the hinge point calibration unit 503 and the independent surroundview image acquisition unit 502, configured to determine the actualrotation angle of the first part relative to the second part, andcombine the actual independent surround view images of the two partsinto a panoramic surround view image of the vehicle according to thecoordinates of the hinge points and the actual rotation angle.

According to another embodiment, the device for generating a panoramicsurround view image of a vehicle may also not include the hinge pointcalibration unit, but receive the coordinates of hinge points fromoutside.

FIG. 6 is a schematic diagram of an independent surround view imageacquisition unit of the device for generating a panoramic surround viewimage of a vehicle according to an embodiment of the present disclosure.As shown in FIG. 6 , the independent surround view image acquisitionunit 502 may include:

An image distortion correction module 61, which is coupled to theoriginal image acquisition module 501, configured to correct distortionon the original images to obtain corrected images corresponding thereto.

A perspective transformation module 62, which is coupled to the imagedistortion correction module 61, configured to project the correctedimages into the geodetic coordinates system to become a correspondingbird's-eye view.

A fixing and stitching module 63, which is coupled to the perspectivetransformation module 62, configured to fix and stitch the bird's-eyeviews of the first part and the second part respectively, to obtain afixed mosaic of the first part and a fixed mosaic of the second partrespectively.

An image cropping module 64, which is coupled to the fixing andstitching module 63, configured to crop the fixed mosaic of the firstpart and the fixed mosaic of the second part to obtain independentsurround view images of the first part and the second part.

The specific method for obtaining the independent surround view imageshas been disclosed in the foregoing embodiment shown in FIG. 3 , whichwill not be repeated here.

FIG. 7 is a schematic diagram of a hinge point calibration unit of thedevice for generating a panoramic surround view image of a vehicleaccording to an embodiment of the present disclosure. As shown in FIG. 7, the hinge point calibration unit 503 may include:

A feature point detecting and matching module 71, which is coupled tothe independent surround view image acquisition unit 502, configured toreceive n pairs of training independent surround view images of thefirst part and the second part, and to detect and match each pair offeature points of training independent surround view images of the firstpart and the second part.

A training rotation angle calculation module 72, which is coupled to thefeature point detecting and matching module 71, configured to obtain ntraining rotation-translation matrices between the feature points ineach pair of training independent surround view images of the first partand the second part based on the coordinates of matched feature points,and accordingly obtain n training rotation angles between the first partand the second part in each pair of independent surround view images.

A training translation vector acquisition module 73, which is coupled tothe training rotation angle calculation module 72, configured todetermine n training translation vectors corresponding to each pair ofindependent surround view images according to the coordinates of thefeature points of each pair of training independent surround view imagesof the first part and the second part and the n training rotationangles.

A hinge point coordinates determination module 74, which is coupled tothe training translation vector acquisition module 73 and the trainingrotation angle calculation module 72, configured to determine thecoordinates of the hinge point of the first part and the second part ofthe vehicle according to the coordinates of the matched feature pointsof the n pairs of training independent surround view images, the ntraining rotation angles and the corresponding n training translationvectors.

The specific method for obtaining the coordinates of hinge points hasbeen disclosed in the embodiment shown in FIG. 4 , which will not berepeated here.

In some embodiments, the hinge point coordinates determination module 74may be further configured to: firstly divide the n pairs of trainingindependent surround view images into in sets where each set has twopairs, and then obtain a calculation result of the corresponding hingepoint coordinates according to the coordinates of the feature points ofeach set of training independent surround view images, the trainingrotation angle and the training translation vector, and then sort thecalculation results of all the sets of coordinates of hinge points afterobtaining the calculation results of the m sets of training independentsurround view images, and take a median of the sorting results as thecoordinates of hinge point, wherein the specific method has beendisclosed in the contents related to the aforementioned formula (5) andformula (6), which will not be repeated here.

In some embodiments, the angle difference of the training rotationangles between the first part and the second part in the set of trainingindependent surround view images may be larger than a preset angle so asto satisfy accuracy of the calculation.

The above-mentioned device for generating a panoramic surround viewimage of a vehicle obtains final coordinates of hinge points by matchingand calculating the feature points of the training independent surroundview images of the two hinged parts. Compared with the traditionalphysical measurement technology, the coordinates of hinge pointsobtained by this method are more accurate. For the vehicles with hingedstructures, this method is of wider adaptability. The method is simpleand reliable in operation, and it can realize the calibration of thehinge points without using other tools such as an angle sensor, therebysaving labor cost and material resources.

FIG. 8 is a schematic diagram of a panoramic surround view imageacquisition unit of a device for generating a panoramic surround viewimage of a vehicle according to an embodiment of the present disclosure.As shown in FIG. 8 , the panoramic surround view image acquisition unit504 includes:

A feature point detecting and matching module 81, which is coupled tothe independent surround view image acquisition unit 502, configured toreceive the actual independent surround view images of the first partand the second part, and to detect and match the feature points therein.81 and 71 may be the same or different modules according to differentembodiments.

An actual rotation angle calculation module 82, which is coupled to thefeature point detecting and matching module 81 and the hinge pointcalibration unit 503, configured to receive the coordinates of hingepoints, and overlap the hinge points of the independent surround viewimages of the first part and the second pan, calculate a distancebetween two points in each matched feature point pair, and take thematched feature point pair with the distance less than the preset firstthreshold as a successfully matched feature point pair, and take therotation angle corresponding to the maximum number of the successfullymatched feature point pairs as a candidate rotation angle of the firstpart relative to the second part.

A panoramic surround view image generation module 83, which is coupledto the actual rotation angle calculation module 82 and the hinge pointcalibration unit 503, configured to use the candidate rotation angle asthe actual rotation angle, and obtain a panoramic surround view image ofthe vehicle after fusing the respective actual independent surround viewimages of the first part and the second part according to thecoordinates of the hinge points and the actual rotation angle.

According to another embodiment, the actual rotation angle calculationmodule 82 may be further configured to determine the coordinates of thesuccessfully matched feature point pair based on the candidate rotationangle, calculate a distance between two points in each successfullymatched feature point pair and sum the distances, and take the rotationangle corresponding to the minimum summation result as the actualrotation angle of the first part relative to the second part

A panoramic surround view image generation module 83, which is coupledto the actual rotation angle calculation module 82 and the hinge pointcalibration unit 503, configured to obtain a panoramic surround viewimage of the vehicle by rotating, translating and stitching therespective actual independent surround view images of the first part andthe second part according to the coordinates of the hinge points and thedetermined actual rotation angle.

FIG. 9 is a schematic diagram of an intelligent vehicle according to anembodiment of the present disclosure. The intelligent vehicle includes aprocessor 901 and a memory 902 for storing a computer program that canbe executed on the processor 901, wherein the processor 901 isconfigured to execute all or part of the methods provided in any of theembodiments of the present disclosure when running the computer program.The respective amount of the processor 901 and the memory 902 herein isnot limited to one, which could be one or more. The intelligent vehiclemay also include an RAM 903, a network interface 904, and a system bus905 connecting the RAM 903, the network interface 904, the processor 901and the memory 902. The operating system and the data processing deviceprovided in the embodiments of the present disclosure are stored in thememory. The processor 901 is used to support the operation of the entireintelligent vehicle. The RAM 903 may be used to provide an environmentfor the execution of computer programs in the memory 902. The networkinterface 904 can be used for external server devices, terminal devicesand the like to perform network communication, receiving or sendingdata, such as obtaining driving control instructions input by users. Theintelligent vehicle may further include a GPS unit 906 configured toobtain location information of the vehicle. The sensor unit 907 mayinclude a wide-angle camera configured to acquire actual or trainingoriginal images.

Embodiments of the present disclosure further provide a computer storagemedium, for example, including a memory storing a computer program,which can be executed by the processor to complete the steps of thecamera attitude information detection provided by any embodiment of thepresent disclosure. The computer storage medium can be such a memory asFRAM, ROM, PROM, EPROM, EEPROM, Flash Memory, magnetic surface memory,optical disk, or CD-ROM; and it can also be various devices includingone or any combination of the above memories.

According to other embodiments, the methods involved in the presentdisclosure may also be executed in whole or in part by a remote serverof an intelligent driving apparatus.

The above-mentioned embodiments only represent several embodiments ofthe present disclosure, and the descriptions thereof are relativelyspecific and detailed, which should not be construed as limiting thescope of the patent application. It should be noted that those skilledin the art could make modifications and improvements without departingfrom the concept of the present disclosure, which all fall into theprotection scope of the present disclosure. Therefore, the scope ofprotection of the present disclosure shall be subject to the appendedclaims.

1. A method for generating a panoramic surround view image of a vehicle,comprising: acquiring actual original images of external environment ofa first part and a second part of the vehicle hinged to each other;processing the actual original images to obtain respective actualindependent surround view images of the first part and the second part;obtaining coordinates of respective hinge points of the first part andthe second part; determining matched feature point pairs in the actualindependent surround view images of the first part and the second part;overlapping the respective hinge points in the actual independentsurround view images of the first part and the second part, and assumingthat the independent surround view image of the first part is rotatedrelative to the independent surround view image of the second part orassuming that matched feature points of the first part are rotatedrelative to matched feature points of the second part, calculating adistance between two points in each matched feature point pairaccordingly, and taking a matched feature point pair with the distanceless than a preset first threshold as a successfully matched featurepoint pair; determining an actual rotation angle of the first partrelative to the second part at least based on number of the successfullymatched feature point pairs; and obtaining a panoramic surround viewimage of the vehicle after fusing the respective actual independentsurround view images of the first part and the second part according tothe coordinates of the hinge points and the actual rotation angle. 2.The method according to claim 1, wherein the step of determining anactual rotation angle of the first part relative to the second part atleast based on number of the successfully matched feature point pairscomprises: taking an angle corresponding to the maximum number of thesuccessfully matched feature point pairs as the actual rotation anglebetween the first part and the second part.
 3. The method according toclaim 1, wherein the step of determining an actual rotation angle of thefirst part relative to the second part at least based on number of thesuccessfully matched feature point pairs comprises: taking an anglecorresponding to the maximum number of the successfully matched featurepoint pairs as a candidate rotation angle between the first part and thesecond part; determining coordinates of the successfully matched featurepoint pairs based on the candidate rotation angle; calculating adistance between two points in each successfully matched feature pointpair and summing the distances; and taking a rotation anglecorresponding to the minimum summation result as the actual rotationangle of the first part relative to the second part.
 4. The methodaccording to claim 3, wherein the step of determining coordinates of thesuccessfully matched feature point pairs based on the candidate rotationangle comprises obtaining a candidate rotation-translation matrix basedon the coordinates of the hinge points of the first part and the secondpart and the candidate rotation angle; and the step of calculating adistance between two points in each successfully matched feature pointpair comprises calculating a distance between two points in thesuccessfully matched feature point pair based on coordinates of thematched feature point pair and the candidate rotation-translationmatrix.
 5. The method according to claim 1, wherein the step ofprocessing the actual original images to obtain respective actualindependent surround view images of the first part and the second partcomprises: correcting distortion on the actual original images of theexternal environment of the first part and the second part; projectingthe corrected images into the geodetic coordinates system to generate abird's-eye view of the first part and the second part; detecting andmatching respectively internal feature points in overlapping areas ofthe respective bird's-eye views of the first part and the second part,and then fixing and stitching to obtain respective fixed mosaics of thefirst part and the second part; and cropping the respective fixedmosaics of the first part and the second part to obtain the respectiveactual independent surround view images of the first part and the secondpart.
 6. The method according to claim 1, wherein the step ofdetermining matched feature point pairs in the actual independentsurround view images of the first part and the second part comprises:feature point detection, for detecting natural feature points in theoverlapping area of the actual independent surround view images of thefirst part and the second part and generating descriptors; feature pointmatching, for generating matched feature point pairs by a matchingalgorithm at least based on the descriptors, wherein the matchingalgorithm comprises ORB, SURF or SIFT algorithm; and feature pointscreening, for screening out mismatched feature point pairs by ascreening algorithm, wherein the screening algorithm comprises RANSAC orGMS algorithm.
 7. The method according to claim 1, wherein obtaining thecoordinates of the respective hinge points of the first part and thesecond part that comprises: obtaining multiple pairs of trainingindependent surround view images of the first part and the second part;detecting and matching feature points in terms of each pair of thetraining independent surround view images; calculating a plurality ofcorresponding training rotation-translation matrices based on thematched feature point pairs in each pair of the training independentsurround view images, and then calculating a plurality of correspondingtraining rotation angles between the first part and the second part;determining a plurality of corresponding training translation vectorsbetween the first part and the second part at least based on coordinatesof the matched feature points in the multiple pairs of trainingindependent surround view images as well as the plurality of trainingrotation angles; and calculating to obtain the coordinates of the hingepoints of the first part and the second part according to thecoordinates of feature points of the multiple pairs of independentsurround view images, the plurality of training rotation angles and theplurality of training translation vectors.
 8. The method according toclaim 7, wherein obtaining the coordinates of the respective hingepoints of the first part and the second part further comprises: takingat least two of the plurality of training rotation angles as one set,and calculating to obtain coordinates of candidate hinge pointscorresponding to the set based on the training translation vectors; andsorting all the coordinates of candidate hinge points, and taking amedian of the sorting result as the coordinates of the hinge points ofthe first part and the second part; wherein a difference between atleast two training rotation angles of each set is greater than a presetangle threshold.
 9. A device for generating a panoramic surround viewimage of a vehicle, the device comprising: an original image acquisitionunit configured to acquire actual or training original images ofexternal environment of a first part and a second part of the vehiclehinged to each other; an independent surround view image acquisitionunit, which is coupled to the original image acquisition unit,configured to stitch the actual or training original images of the firstpart and the second part into respective actual or training independentsurround view images; and a panoramic surround view image acquisitionunit coupled to a hinge point calibration unit and the independentsurround view image acquisition unit, comprising: a feature pointdetecting and matching module, which is coupled to the independentsurround view image acquisition unit, configured to receive the actualindependent surround view images of the first part and the second part,and detect and match feature points therein; an actual rotation anglecalculation module, which is coupled to the feature point detecting andmatching module, configured to obtain coordinates of hinge points of thefirst part and the second part, and overlap the hinge points of thefirst part and the second part in the independent surround view images,and, assuming that the independent surround view image of the first partis rotated relative to the independent surround view image of the secondpart or that the matched feature points of the first part are rotatedrelative to the matched feature points of the second part, calculate adistance between two points in each matched feature point pair to take amatched feature point pair with the distance less than a preset firstthreshold as a successfully matched feature point pair, and determine anactual rotation angle of the first part relative to the second part atleast based on number of the successfully matched feature point pairs;and a panoramic surround view image generation module, which is coupledto the actual rotation angle calculation module, configured to obtain apanoramic surround view image of the vehicle after fusing the respectiveactual independent surround view images of the first part and the secondpart according to the coordinates of the hinge points and the actualrotation angle.
 10. The device according to claim 9, wherein the actualrotation angle calculation module configured to determine the actualrotation angle of the first part relative to the second part at leastbased on the number of the successfully matched feature point pairscomprises, taking an angle corresponding to the maximum number of thesuccessfully matched feature point pairs as the actual rotation anglebetween the first part and the second part.
 11. The device according toclaim 9, wherein the actual rotation angle calculation module configuredto determine the actual rotation angle of the first part relative to thesecond part at least based on the number of the successfully matchedfeature point pairs comprises, taking an angle corresponding to themaximum number of the successfully matched feature point pairs as acandidate rotation angle between the first part and the second part,determining coordinates of the successfully matched feature point pairbased on the candidate rotation angle, calculating a distance betweentwo points in each successfully matched feature point pair and summingthe distances, and taking a rotation angle corresponding to the minimumsummation result as the actual rotation angle of the first part relativeto the second part.
 12. The device according to claim 9, furthercomprising the hinge point calibration unit coupled to the independentsurround view image acquisition unit, which comprises: a feature pointdetecting and matching module, which is coupled to the independentsurround view image acquisition unit, configured to receive multiplepairs of training independent surround view images of the first part andsecond part, and to detect and match feature points of each pair oftraining independent surround view images of the first part and thesecond part; a training rotation angle calculation module, which iscoupled to the feature point detecting and matching module, configuredto obtain a plurality of training rotation-translation matrices betweenthe feature points of the first part and the second part in each pair oftraining independent surround view images based on the coordinates ofthe matched feature points, and correspondingly obtain a plurality oftraining rotation angles between the first part and the second part ineach pair of independent surround view images; a training translationvector acquisition module, which is coupled to the training rotationangle calculation module, configured to determine a plurality ofcorresponding training translation vectors of each pair of trainingindependent surround view images according to coordinates of the matchedfeature points of each pair of training independent surround view imagesand the plurality of training rotation angles; and a hinge pointcoordinates determination module, which is coupled to the translationvector acquisition module and the training rotation angle calculationmodule, configured to determine the coordinates of the hinge points ofthe first part and the second part of the vehicle according to thecoordinates of the matched feature points of the multiple pairs oftraining independent surround view images, the plurality of trainingrotation angles and the corresponding plurality of training translationvectors.
 13. An intelligent vehicle, comprising a first part and asecond part hinged to each other; a processor, and a memory coupled tothe processor; and a sensor unit configured to capture actual ortraining original images of the first part and the second part; whereinthe processor is configured to implement the method according to claim1.